booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted #1676

rvagg · 2023-09-07T03:29:05Z

This is a bit of a PoC at the moment; enables a mini Trustless Gateway implementation when you only want to serve CARs through /ipfs/. Uses Frisbii but I've copied in the two main modules that are used into this PR because of the libp2p dependency pain that gets stuck at ipni dependencies in Frisbii which are not required here. So cmd/booster-http/frisbii/* are (mostly) unmodified copies from github.com/ipld/frisbii - we only need the handler and the CAR streamer for this.

The transparent switch in implementations by subtle CLI argument changes is probably not optimal for users, but I've got a big integration test in here that pulls in the big trustless pathing test fixture that exercises a whole lot of trustless path retrievals for a fixed 20Mb CAR (which we import across multiple deals here) and it gets run across both implementations in here with the same boost backend, just two different booster-http frontends.

The reason this might be useful are that (1) it's super minimal and very lightweight; there's no intermediate caching of blocks while it builds up state, it just does a traversal straight from the backend blockstore and spits it out as a CAR, (2) it gives us a bit more freedom to experiment with alternative outputs, like what SPARK is after, and (3) gives us useful error output if there's a problem loading the root block, instead of the current empty CAR that we get.

Update docs

dirkmc · 2023-09-07T11:47:31Z

Thanks for putting this together @rvagg

We actually have a PR that simplifies the parameters to booster-http, and takes advantage of the boxo gateway's trustless gateway: feat: add support for deserialized response

How would pulling in this frisbii implementation differ from using the boxo gateway?

willscott · 2023-10-02T10:31:30Z

This direction was well-received at the fil dev summit last week.

having a separate bifrost-gateway port / process for full files, assuming we have documentation for that install, allows the two endpoints to have different ACLs
not having the full boxo dependencies directly in booster-http lets us be a bit less worried about dependency versioning.

@rvagg - what's needed to get this to a point where it could be ready for review?

rvagg · 2023-10-03T04:09:31Z

Changed so this removes --serve-blocks and --serve-files, leaving only --serve-pieces and --serve-cars - http piece, and http trustless.

The whole frisbii directory will go when the ipnisync branch gets merged and we get all the updated dependencies. For now it's an up to date fork of the frisbii components being used.

rvagg · 2023-10-05T02:54:33Z

removed the frisbii fork now that the libipni branch is merged and dependencies updated so we have no conflicts!

LexLuthr · 2023-10-05T08:09:33Z

Does this require docs changes? If yes, then can we add some rough documentation to this PR and I can update boost docs.

rvagg · 2023-10-06T04:43:20Z

yes, working on it, first I'll do so by getting an integration test in here that uses bifrost-gateway to demonstrate that it can be used on top of this, then we can add that to the docs as well so all options are covered.

rvagg · 2023-10-06T10:14:10Z

includes a full e2e test now that involves bifrost-gateway and lassie to perform some trustless and trusted fetching, these are both installed on demand and are not direct dependencies; I have some cleanup to do before this is ready but it's mostly there now

rvagg · 2023-10-10T03:20:59Z

Rebased and merged in #1736 and #1737 to this and simplified the commits significantly. There's 4 commits now - the main switch to frisbii, the fixes for compression and addition of --compression-level, the addition of --log-file and removal of custom logging in here, and the refactor of the piece handler to a separate file.

The bulk of the changes here are the addition of extra tests, including an e2e one involving bifrost-gateway to exercise trusted gateway support, and the refactoring of the piece handler does create some annoying diff noise, but it's last in the commit list so you could review that separately to avoid the noise—there are a bunch of changes to the piece handler to support logging and compression that are masked by the refactor.

Further comments about compression are in #1736 and logging, including example output, is in #1737.

cmd/booster-http/e2e_test.go

cmd/booster-http/http_test.go

cmd/booster-http/piecehandler.go

cmd/booster-http/run.go

rvagg · 2023-10-12T03:56:59Z

Updated to frisbii@v0.4.1 for a minor byte-range fix but this also brings in some other deps - notably graphsync v0.15, which thankfully doesn't cause problems here now because of the libp2p update (which was the main difference between the head versions on v0.14 and v0.15) and go-graphsync is only used for testing, with boost-graphsync used for builds here.

Also brings in various golang.org/x/ upgrades which come along with golang.org/x/net@v0.17.0 for the http/2 security fixes ipld/frisbii#63

LexLuthr · 2023-10-18T21:17:18Z

Pending AI to merge in main:

Rebase on the latest main
Test in devnet for multi-miner retrievals
Test in prod for HTTP retrievals
Perf test to compare with the older HTTP server

* print info msgs on removed options * add full e2e+bifrost-gateway test

… GzipHandler usage

…d http log output

LexLuthr · 2023-10-25T11:11:38Z

DEVNET Testing:

lexluthr@Lexs-Mac ~ % curl -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/piece/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8128k  100 8128k    0     0   100M      0 --:--:-- --:--:-- --:--:--  107M
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/piece/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /piece/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: public, max-age=29030400, immutable
< Content-Length: 8323072
< Content-Type: application/piece
< Etag: "baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy"
< Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
< Vary: Accept-Encoding
< Date: Wed, 25 Oct 2023 11:03:45 GMT
< 
{ [13901 bytes data]
100 8128k  100 8128k    0     0   107M      0 --:--:-- --:--:-- --:--:--  115M
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % l              
zsh: command not found: l
lexluthr@Lexs-Mac ~ % la
zsh: command not found: la
lexluthr@Lexs-Mac ~ % ls b
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
8.1M	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< Vary: Origin
< Vary: Accept-Encoding
< Date: Wed, 25 Oct 2023 11:04:25 GMT
< Content-Length: 135
< Content-Type: text/plain; charset=utf-8
< 
{ [135 bytes data]
100   135  100   135    0     0  15180      0 --:--:-- --:--:-- --:--:-- 33750
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car                                                                                                
4.0K	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea.car"
< Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
< Etag: "bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea.car.bjhgn9hgie2db"
< Vary: Accept, Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafk2bzaceaaggikxeo6icjpcfjqkpi3ziarfa3ej6tg3wljkmyyvwgetiv7ea
< Date: Wed, 25 Oct 2023 11:04:52 GMT
< Transfer-Encoding: chunked
< 
{ [13680 bytes data]
100 51302    0 51302    0     0  1141k      0 --:--:-- --:--:-- --:--:-- 1318k
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car                                                                                              
52K	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76 HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76.car"
< Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
< Etag: "bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76.car.9e13ncrk162k"
< Vary: Accept, Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76
< Date: Wed, 25 Oct 2023 11:05:24 GMT
< Transfer-Encoding: chunked
< 
{ [3545 bytes data]
100 5009k    0 5009k    0     0  8289k      0 --:--:-- --:--:-- --:--:-- 8348k
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car                                                                                              
5.1M	baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car

LexLuthr · 2023-10-25T11:58:30Z

Tested on filcollins:

lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqf2rvj4heoov5cwf76hshbxu3747x3dlce3sxqebnvfblrcbqaacy.car http://localhost:7777/ipfs/bafykbzacebujlhyuu4l73ulhjrg4nlwdgdb6nwyzfrssn7m4rkbyevmq24r76
lexluthr@Lexs-Mac ~ % curl -v -o baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai http://localhost:7777/piece/baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /piece/baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: public, max-age=29030400, immutable
< Content-Length: 34091302912
< Content-Type: application/piece
< Etag: "baga6ea4seaqpx3gpqcqjqnw6zent5wso6zvgkvzdg34iktuu66naio72emcnmai"
< Last-Modified: Thu, 01 Jan 1970 00:00:00 GMT
< Vary: Accept-Encoding
< Date: Wed, 25 Oct 2023 11:55:08 GMT
< 
{ [3761 bytes data]
  0 31.7G    0 27.3M    0     0  3753k      0  2:27:50  0:00:07  2:27:43 5498k^C
lexluthr@Lexs-Mac ~ % 
lexluthr@Lexs-Mac ~ % 
lexluthr@Lexs-Mac ~ % 
lexluthr@Lexs-Mac ~ % curl -v -o bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca http://localhost:7777/piece/bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /piece/bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Vary: Origin
< Vary: Accept-Encoding
< X-Content-Type-Options: nosniff
< Date: Wed, 25 Oct 2023 11:56:25 GMT
< Content-Length: 103
< 
{ [103 bytes data]
100   103  100   103    0     0    179      0 --:--:-- --:--:-- --:--:--   180
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca 
4.0K	bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca
lexluthr@Lexs-Mac ~ % cat bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca 
"getting piece metadata: piece bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca: not found"
lexluthr@Lexs-Mac ~ % curl -v -o bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca http://localhost:7777/ipfs/bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 127.0.0.1:7777...
* Connected to localhost (127.0.0.1) port 7777 (#0)
> GET /ipfs/bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi HTTP/1.1
> Host: localhost:7777
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Cache-Control: public, max-age=29030400, immutable
< Content-Disposition: attachment; filename="bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi.car"
< Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=y
< Etag: "bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi.car.76dbvvp97jlgt"
< Vary: Accept, Accept-Encoding
< X-Content-Type-Options: nosniff
< X-Ipfs-Path: /ipfs/bafkreiaacexznag7sac257mh56cvmicyqjqxue26jwfxja4marl6h4nxzi
< Date: Wed, 25 Oct 2023 11:57:17 GMT
< Transfer-Encoding: chunked
< 
{ [32225 bytes data]
100 1024k    0 1024k    0     0   429k      0 --:--:--  0:00:02 --:--:--  429k
* Connection #0 to host localhost left intact
lexluthr@Lexs-Mac ~ % du -sh bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca                                                                                           
1.1M	bafkreiaacvjytvl52wrmwndtt56ibztwqebybyluw5gznqmru33mvpssca

LexLuthr

All tests are complete. I added some fixes for devnet.
I think we are good to merge.

rvagg marked this pull request as draft September 7, 2023 03:29

rvagg force-pushed the rvagg/frisbii branch 3 times, most recently from 7a42013 to 4ef62b7 Compare October 3, 2023 04:01

rvagg marked this pull request as ready for review October 3, 2023 04:09

rvagg force-pushed the rvagg/frisbii branch from 4ef62b7 to e8103a2 Compare October 5, 2023 02:30

rvagg force-pushed the rvagg/frisbii branch from a056c4c to ed3814b Compare October 6, 2023 10:13

rvagg force-pushed the rvagg/frisbii branch 2 times, most recently from 4e4b295 to 3b07968 Compare October 9, 2023 02:37

This was referenced Oct 9, 2023

feat(booster-http): --compression-level flag for piece & frisbii, fix GzipHandler usage #1736

Closed

feat(booster-http): --log-file, defaulting to stdout, for standardised http log output #1737

Closed

nonsense requested review from nonsense and masih October 9, 2023 15:20

rvagg force-pushed the rvagg/frisbii branch from c94a6aa to 152614a Compare October 10, 2023 03:14

rvagg changed the title ~~feat: use mini gateway impl when only --serve-cars~~ booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted Oct 10, 2023

masih approved these changes Oct 10, 2023

View reviewed changes

rvagg force-pushed the rvagg/frisbii branch 2 times, most recently from 7a9f439 to 95a411c Compare October 12, 2023 03:52

nonsense approved these changes Oct 16, 2023

View reviewed changes

rvagg force-pushed the rvagg/frisbii branch from 95a411c to c86fa23 Compare October 19, 2023 00:00

rvagg added 6 commits October 25, 2023 12:10

feat(booster-http)!: remove full gateway, only offer --serve-cars

2951364

* print info msgs on removed options * add full e2e+bifrost-gateway test

feat(booster-http): --compression-level flag for piece & frisbii, fix…

a586156

… GzipHandler usage

feat(booster-http): --log-file, defaulting to stdout, for standardise…

21ec183

…d http log output

chore(booster-http): split out piece handler code to separate file

09b41cb

fix: address review feedback

57d7e98

chore(deps): update frisbii for entity-bytes=0:0 fix

577abc9

rvagg force-pushed the rvagg/frisbii branch from c86fa23 to 577abc9 Compare October 25, 2023 01:10

fix devnet

1e69ade

LexLuthr approved these changes Oct 25, 2023

View reviewed changes

LexLuthr merged commit 7b92e23 into main Oct 26, 2023
21 checks passed

LexLuthr deleted the rvagg/frisbii branch October 26, 2023 09:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted #1676

booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted #1676

rvagg commented Sep 7, 2023 •

edited by LexLuthr

Loading

dirkmc commented Sep 7, 2023

willscott commented Oct 2, 2023

rvagg commented Oct 3, 2023

rvagg commented Oct 5, 2023

LexLuthr commented Oct 5, 2023

rvagg commented Oct 6, 2023

rvagg commented Oct 6, 2023

rvagg commented Oct 10, 2023

rvagg commented Oct 12, 2023 •

edited

Loading

LexLuthr commented Oct 18, 2023 •

edited

Loading

LexLuthr commented Oct 25, 2023

LexLuthr commented Oct 25, 2023

LexLuthr left a comment

booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted #1676

booster-http refactor - trustless only, improved compression, logging & e2e testing w/ trusted #1676

Conversation

rvagg commented Sep 7, 2023 • edited by LexLuthr Loading

dirkmc commented Sep 7, 2023

willscott commented Oct 2, 2023

rvagg commented Oct 3, 2023

rvagg commented Oct 5, 2023

LexLuthr commented Oct 5, 2023

rvagg commented Oct 6, 2023

rvagg commented Oct 6, 2023

rvagg commented Oct 10, 2023

rvagg commented Oct 12, 2023 • edited Loading

LexLuthr commented Oct 18, 2023 • edited Loading

LexLuthr commented Oct 25, 2023

LexLuthr commented Oct 25, 2023

LexLuthr left a comment

Choose a reason for hiding this comment

rvagg commented Sep 7, 2023 •

edited by LexLuthr

Loading

rvagg commented Oct 12, 2023 •

edited

Loading

LexLuthr commented Oct 18, 2023 •

edited

Loading